Image processor, image capture device, image processing method and program

ABSTRACT

This image processor  10  includes: an input interface  11  that receives first and second images, on one of which the position of the subject has shifted in a particular direction from the position on the other; a color image generating section  13   a  that generates a color image in which the pixel values of respective pixels of the first and second images are used as the values of first and second colors, respectively; a decision section  13   b  that calculates an index value indicating the degree of color shift between the first and second colors in the color image and that determines, based on the index value, whether or not the first and second images match each other; and an image moving section  13   c  that performs, if the decision has been made that the first and second images do not match each other, the processing of making the second image closer to the first image by replacing the pixel value of each pixel of the second image with the pixel value of a pixel that is adjacent to the former pixel in the particular direction.

TECHNICAL FIELD

The present application relates to a single-lens 3D image capturingtechnology for obtaining subject's depth information using a singleoptical system and a single image sensor.

BACKGROUND ART

Recently, the performance and functionality of digital cameras anddigital movie cameras that use some solid-state image sensor such as aCCD or a CMOS (which will be sometimes simply referred to herein as an“image sensor”) have been enhanced to an astonishing degree. Inparticular, the size of a pixel structure for use in an image sensor hasbeen further reduced these days thanks to rapid development ofsemiconductor device processing technologies, thus getting an evengreater number of pixels and drivers integrated together in an imagesensor. As a result, the resolution of an image sensor has latelyincreased rapidly from around one million pixels to ten million or morepixels in a matter of few years. On top of that, the quality of an imagecaptured has also been improved significantly as well. As for displaydevices, on the other hand, LCD and plasma displays with a reduced depthnow provide high-resolution and high-contrast images, thus realizinghigh performance without taking up too much space. And such videoquality improvement trends are now spreading from 2D images to 3Dimages. In fact, 3D display devices that achieve high image qualityalthough they require the viewer to wear a pair of polarization glasseshave been developed just recently.

As for the 3D image capturing technology, a typical 3D image capturedevice with a simple arrangement uses an image capturing system with twocameras to capture a right-eye image and a left-eye image. According tosuch a so-called “two-lens image capturing” technique, however, twocameras need to be used, thus increasing not only the overall size ofthe image capture device but also the manufacturing cost as well. Toovercome such a problem, methods for capturing multiple images withparallax (which will be sometimes referred to herein as “multi-viewpointimages”) by using a single camera have been researched and developed.Such a method is called a “single-lens image capturing method”.

For example, Patent Documents Nos. 1 and 2 disclose a method forobtaining multi-viewpoint images by using two polarizers, of which thetransmission axes cross each other at right angles, and a rotatingpolarization filter. Meanwhile, Patent Documents Nos. 3 to 5 disclose amethod for obtaining multi-viewpoint images by using a diaphragm (lightbeam confining plate) with multiple color filters.

The methods disclosed in these Patent Documents Nos. 1 to 5 are usedmostly to generate multi-viewpoint images using a single-lens camera. Onthe other hand, there is a technique for getting depth information usinga single-lens camera with multiple micro lenses and for changing thefocus position of the image captured freely based on that information.Such a technique is called “light field photography” and a single-lenscamera that uses such a technique is called a “light field camera”. In alight field camera, a number of micro lenses are arranged on an imagesensor. Each of those micro lenses is arranged so as to cover aplurality of pixels. By calculating information about the direction ofincoming light based on the image information gotten through the imagecapturing session, the subject's depth can be estimated. Such a camerais disclosed in Non-Patent Document No. 1, for example.

The light field camera can calculate depth information. But itsresolution is determined by the number of micro lenses and should belower than the resolution determined by the number of pixels of theimage sensor, which is a problem. Thus, to overcome such a problem,Patent Document No. 6 discloses a technique for increasing theresolution using two image capturing systems. According to such atechnique, the incoming light is split into two divided incoming lightbeams, which are imaged by two image capturing systems, of which thegroups of micro lenses are arranged so as to spatially shift from eachother by a half pitch, and then the images captured in this manner aresynthesized together, thereby increasing the resolution. However, thistechnique requires two image capturing systems, thus causing size andcost problems, too.

To overcome such a problem, Patent Document No. 7 discloses a techniquefor changing the modes of operation from a normal shooting mode into thelight field photography mode, or vice versa, using a single imagecapturing system. According to this technique, a micro lens, of whichthe focal length varies according to the voltage applied, is used.Specifically, the focal length of the micro lens is set to be theinfinity in the former mode and set to be a predetermined length in thelatter mode. By adopting such a mechanism, an image with high resolutionand depth information can be obtained. However, this technique requiresa sophisticated control technique for controlling the focal length ofthe micro lens.

Meanwhile, Patent Documents Nos. 8 and 9 disclose techniques which weredeveloped mainly for the purpose of getting depth information. Accordingto these techniques, an image is captured through a diffraction gratingwhich is arranged in front of a camera, and the distance from thesubject to the diffraction grating is measured based on the magnitude ofpositional shift between an image produced by a zero-order diffractedlight beam that has been transmitted through the diffraction grating(which will be referred to herein as a “zero-order light image”) and animage produced by a high-order diffracted light beam (which will bereferred to herein as a “high-order light image”)

CITATION LIST Patent Literature

-   Patent Document No. 1: Japanese Laid-Open Patent Publication No.    62-291292-   Patent Document No. 2: Japanese Laid-Open Patent Publication No.    62-217790-   Patent Document No. 3: Japanese Laid-Open Patent Publication No.    2002-344999-   Patent Document No. 4: Japanese Laid-Open Patent Publication No.    2009-276294-   Patent Document No. 5: Japanese Laid-Open Patent Publication No.    2003-134533-   Patent Document No. 6: Japanese Laid-Open Patent Publication No.    11-98532-   Patent Document No. 7: Japanese Laid-Open Patent Publication No.    2008-167395-   Patent Document No. 8: PCT International Application Japanese    National Phase Publication No. 2-502398-   Patent Document No. 9: Japanese Laid-Open Patent Publication No.    2011-2387

Non-Patent Literature

-   Non-Patent Document No. 1: Ren Ng, et al., “Light Field Photography    with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR    2005-02

SUMMARY OF INVENTION Technical Problem

With the conventional light field camera, depth information can becertainly obtained but the resolution of the resultant image decreases,which is a problem. To overcome the problem, the optical system shouldbe modified as in the techniques disclosed in Patent Documents Nos. 6and 7. To modify the optical system, however, either two image capturingsystems should be used or the focus length of the micro lens should becontrolled.

Also, according to the method for measuring the depth using adiffraction grating as disclosed in Patent Documents Nos. 8 and 9, it isso difficult to separate the zero-order light image and the high-orderlight image from each other that the distance to the subject sometimescannot be measured accurately. For example, according to the methoddisclosed in Patent Document No. 9, a zero-order light image and afirst-order light image are distinguished based on the luminancegradient of a diffracted image and the magnitude of their shift ismeasured. And the subject's depth is obtained based on that magnitude ofshift. Actually, however, it is difficult to accurately match two imageswith mutually different luminance levels such as the zero-order lightimage and the first-order light image.

An embodiment of the present invention provides an image capturingtechnique, by which an image with a minimized decrease in resolution anddepth information can be obtained at the same time using a differentoptical system and a different kind of signal processing than theconventional ones. Another embodiment of the present invention providesan image processing technique, by which multiple images including thesame subject but having mutually different luminance levels can beeasily matched to each other.

Solution to Problem

To overcome the problem described above, an image processor as anembodiment of the present invention carries out matching on a pluralityof images representing the same subject. The processor includes: aninput interface that receives first and second images on one of whichthe position of the subject has shifted in a particular direction fromthe position on the other; a color image generating section thatgenerates a color image in which the pixel values of respective pixelsof the first and second images are used as the values of first andsecond colors, respectively; a decision section that calculates an indexvalue indicating the degree of color shift between the first and secondcolors in the color image and that determines, based on the index value,whether or not the first and second images match each other; and animage moving section that performs, if the decision has been made thatthe first and second images do not match each other, the processing ofmaking the second image closer to the first image by replacing the pixelvalue of each pixel of the second image with the pixel value of a pixelthat is adjacent to the former pixel in the particular direction.

This general and particular aspect can be implemented as a system, amethod, a computer program or a combination thereof.

Advantageous Effects Of Invention

According to an embodiment of the present invention, an image producedby diffracted light and an image produced by straight light can beseparated from each other, and therefore, the subject's depth can becalculated based on those images. Also, according to another embodiment,a plurality of images representing the same subject can be easilymatched to each other.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A diagram illustrating a configuration for an image capturedevice according to an exemplary embodiment.

FIG. 2 A schematic representation illustrating an imaging area accordingto an exemplary embodiment.

FIG. 3A A plan view illustrating a light-transmitting plate according toa first exemplary embodiment.

FIG. 3B A cross-sectional view as viewed on the plane A-A′ shown in FIG.3A.

FIG. 3C A cross-sectional view as viewed on the plane B-B′ shown in FIG.3A.

FIG. 4 Illustrates a basic pixel arrangement for an image sensoraccording to the first exemplary embodiment.

FIG. 5 A plan view illustrating an image sensor according to the firstexemplary embodiment.

FIG. 6A A schematic representation illustrating an example of imagesobtained by capturing according to the first exemplary embodiment.

FIG. 6B A schematic representation illustrating a state where twodiffracted light images are shifted toward a direct light image.

FIG. 6C A schematic representation illustrating a state where the twodiffracted light images are further shifted toward, and substantiallysuperposed on, the direct light image.

FIG. 7A A flowchart showing a procedure of depth information generationprocessing according to the first exemplary embodiment.

FIG. 7B A flowchart showing an alternative procedure of depthinformation generation processing according to the first exemplaryembodiment.

FIG. 8 A graph showing an exemplary relation between the pixel shiftnumber and the degree of R/B coloring according to the first exemplaryembodiment.

FIG. 9A A plan view illustrating a modified example of thelight-transmitting plate according to the first exemplary embodiment.

FIG. 9B A plan view illustrating another modified example of thelight-transmitting plate according to the first exemplary embodiment.

FIG. 10A A plan view illustrating a light-transmitting plate accordingto a second exemplary embodiment.

FIG. 10B A cross-sectional view as viewed on the plane C-C′ shown inFIG. 10A.

FIG. 11 A block diagram illustrating an exemplary configuration for animage processor according to a third exemplary embodiment.

FIG. 12 A schematic representation illustrating conceptually how to getmatching processing done according to the third exemplary embodiment.

FIG. 13 A flowchart showing the procedure of the matching processingaccording to the third exemplary embodiment.

FIG. 14 A block diagram illustrating another exemplary configuration foran image processor according to the third exemplary embodiment.

DESCRIPTION OF EMBODIMENTS

Exemplary embodiments of the present invention are outlined as follows:

An image processor as an embodiment of the present invention carries outmatching on a plurality of images representing the same subject, andincludes: an input interface that receives first and second images onone of which the position of the subject has shifted in a particulardirection from the position on the other; a color image generatingsection that generates a color image in which the pixel values ofrespective pixels of the first and second images are used as the valuesof first and second colors, respectively; a decision section thatcalculates an index value indicating the degree of color shift betweenthe first and second colors in the color image and that determines,based on the index value, whether or not the first and second imagesmatch each other; and an image moving section that performs, if thedecision has been made that the first and second images do not matcheach other, the processing of making the second image closer to thefirst image by replacing the pixel value of each pixel of the secondimage with the pixel value of a pixel that is adjacent to the formerpixel in the particular direction.

(2) In one embodiment of the image processor of (1), if the image movingsection has performed the processing, the decision section determinesagain whether or not the first and second images match each other, andthe image moving section performs the processing over and over againuntil the decision is made by the decision section that the first andsecond images match each other.

(3) In one embodiment of the image processor of (1) or (2), the decisionsection adjusts the balance between the respective values of the firstand second colors in multiple pixels of the color image, and obtains theindex value by performing an arithmetic operation including calculatingthe difference between the respective values of the first and secondcolors that have been adjusted in each pixel.

(4) In one embodiment of the image processor of (3), the index value isobtained by adding together either the absolute values, or the squares,of the differences between the respective values of the first and secondcolors that have been adjusted with respect to every pixel.

(5) In one embodiment of the image processor of (4), if it has turnedout, as a result of the processing by the image moving section, that theindex value has increased from the previous one, the decision is made bythe decision section that the first and second images match each other.

(6) In one embodiment, the image processor of one of (1) to (5) furtherincludes an image dividing section that divides, if the decision hasbeen made that the first and second images match each other, each of thefirst and second images into a plurality of partial images. Whilechanging combinations of the partial images, one of which has beenselected from the first image and the other of which has been selectedfrom the second image, the decision section calculates the index valuewith respect to an area of the color image associated with thecombination of the partial images, thereby choosing a combination of thepartial images that have the highest degree of matching. The imagemoving section makes the second image even closer to the first imagebased on a difference in coordinate between the partial images in thecombination that has been chosen by the decision section.

(7) In one embodiment of the image processor of one of (1) to (6), thefirst color is one of the colors red, green and blue and the secondcolor is the complementary color of the first color.

In one embodiment of the image processor of one of (1) to (6), the inputinterface further obtains a third image representing the same subject asthe first and second images. If the respective positions of the subjecton the second and third images are symmetric to the position of thesubject on the first image, the color image generating section generatesthe color image in which the respective pixel values of the first,second and third images are used as the values of the first, second andthird colors, respectively. The decision section calculates an indexvalue indicating the degree of color shift between the first, second andthird colors and determines, based on the index value, whether or notthe second and third images match the first image. If the decision hasbeen made that the second and third images do not match the first image,the image moving section performs the processing of making the secondand third images closer to the first image by replacing the pixel valueof each pixel of the second image with the pixel value of a pixel thatis adjacent to the former pixel in a first direction and by replacingthe pixel value of each pixel of the third image with the pixel value ofa pixel that is adjacent to the former pixel in a direction opposite tothe first direction.

(9) In one embodiment of the image processor of (8), the first, secondand third colors are respectively one, another and the other of thecolors red, green and blue.

(10) In one embodiment, the image processor of one of (1) to (9) furtherincludes an output interface that outputs information about themagnitude of overall motion of the second image as a result of theprocessing by the image moving section.

(11) An image capture device as an embodiment of the present inventionincludes: the image processor of one of (1) to (10); and an imagecapturing section that obtains the first and second images by capturing.

(12) An image processing method according to an embodiment of thepresent invention is designed to carry out matching on a plurality ofimages representing the same subject. The method includes the steps of:receiving first and second images, on one of which the position of thesubject has shifted in a particular direction from the position on theother; generating a color image in which the pixel values of respectivepixels of the first and second images are used as the values of firstand second colors, respectively; calculating an index value indicatingthe degree of color shift between the first and second colors in thecolor image; determining, based on the index value, whether or not thefirst and second images match each other; and if the decision has beenmade that the first and second images do not match each other, makingthe second image closer to the first image by replacing the pixel valueof each pixel of the second image with the pixel value of a pixel thatis adjacent to the former pixel in the particular direction.

(13) A program as an embodiment of the present invention is designed tocarry out matching on a plurality of images representing the samesubject. The program is defined to make a computer perform the steps of:receiving first and second images, on one of which the position of thesubject has shifted in a particular direction from the position on theother; generating a color image in which the pixel values of respectivepixels of the first and second images are used as the values of firstand second colors, respectively; calculating an index value indicatingthe degree of color shift between the first and second colors in thecolor image; determining, based on the index value, whether or not thefirst and second images match each other; and if the decision has beenmade that the first and second images do not match each other, makingthe second image closer to the first image by replacing the pixel valueof each pixel of the second image with the pixel value of a pixel thatis adjacent to the former pixel in the particular direction.

Hereinafter, more specific embodiments of the present invention will bedescribed with reference to the accompanying drawings. In the followingdescription, any element shown in multiple drawings and havingsubstantially the same function will be identified by the same referencenumeral.

Embodiment 1

First of all, a depth estimating image capture device as firstembodiment of the present invention will be described. FIG. 1 is a blockdiagram illustrating an overall configuration for an image capturedevice according to this embodiment. The image capture device of thisembodiment is a digital electronic camera and includes an imagecapturing section 100 and a signal processing section 200 that generatesa signal representing an image (i.e., an image signal) based on thesignal generated by the image capturing section 100. The image capturedevice may have the function of generating a moving picture, not just astill picture.

The image capturing section 100 includes a solid-state image sensor 2(which will be simply referred to herein as an “image sensor”) with anumber of photosensitive cells which are arranged on its imaging area(and which will be sometimes referred to herein as “pixels”), alight-transmitting plate (light-transmitting section) 1 with adiffraction grating area and a polarization area, an optical lens(imaging section) 3 which produces an image on the imaging area of theimage sensor 2, and an infrared cut filter 4. Even though thelight-transmitting plate 1 is arranged in front of the optical lens 3,the light-transmitting plate 1 may also be arranged behind the opticallens 3 as well. The image sensor 2 includes pixels on which polarizationfilters are arranged and pixels on which no polarization filters arearranged. The image capturing section 100 further includes a signalgenerating and receiving section 5 which not only generates afundamental signal to drive the image sensor 2 but also receives theoutput signal of the image sensor 2 and sends it to the signalprocessing section 200, and a sensor driving section 6 which drives theimage sensor 2 in accordance with the fundamental signal generated bythe signal generating and receiving section 5. The image sensor 2 istypically a CCD or CMOS sensor, which may be fabricated by knownsemiconductor device processing technologies. The signal generating andreceiving section 5 and the sensor driving section 6 may be implementedas an LSI such as a CCD driver.

The signal processing section 200 includes an image processing section 7which processes the output signal of the image capturing section 100 togenerate a plurality of images, a memory 30 which stores various kindsof data for use to generate the image signal, and an image interface(I/F) section 8 which sends out the image signal and depth informationthus generated to an external device. The image processing section 7includes an image generating section 7 a which generates imageinformation based on the signal supplied from the image capturingsection 100 and a depth information generating section 7 b whichgenerates depth information. The image processing section 7 may be acombination of a hardware component such as a known digital signalprocessor (DSP) and a software program for use to perform imageprocessing involving the image signal generation. The memory 30 may be aDRAM, for example. And the memory 30 not only stores the signal suppliedfrom the image capturing section 100 but also temporarily retains theimage data that has been generated by the image processing section 7 orcompressed image data. These image data are then output to either astorage medium or a display section by way of the interface section 8.

The depth estimating image capture device of this embodiment actuallyfurther includes an electronic shutter, a viewfinder, a power supply (orbattery), a flashlight and other known components. However, descriptionthereof will be omitted herein, because none of them are essentialcomponents that should be described in detail to allow the reader tounderstand how the present invention works. Also, this configuration isonly an example. Thus, in this embodiment, additional components otherthan the light-transmitting plate 1, the image sensor 2 and the imageprocessing section 7 may be implemented as an appropriate combination ofknown elements.

Hereinafter, the configuration of the image capturing section 100 willbe described in detail. In the following description, when a position ordirection in the image capturing area is mentioned, the xy coordinatesshown in the drawings will be used.

FIG. 2 schematically illustrates the relative arrangement of the opticallens 3, infrared cut filter 4, light-transmitting plate 1, and imagesensor 2 in the image capturing section 100. The optical lens 3 does nothave to be a single lens but may be a lens unit comprised of groups oflenses. But the optical lens 3 is drawn in FIG. 2 as a single lens forthe sake of simplicity. The optical lens 3 is a known lens and condensesthe incoming light and images the light on the image capturing sectionof the image sensor 2. It should be noted that the relative arrangementof the respective members shown in FIG. 2 is only an example and doesnot always have to be adopted according to the present invention. Forexample, there is no problem if the positions of the optical lens 3 andthe infrared cut filter 4 are exchanged with each other. Also, theoptical lens 3 and the light-transmitting plate 1 may be integrated witheach other. Furthermore, although the infrared cut filter 4 is providedaccording to this embodiment because a visible radiation image issupposed to be captured, the infrared cut filter 4 would be unnecessaryin an application in which an infrared image needs to be obtained (e.g.,in a night surveillance camera).

FIG. 3A is a plan view of the light-transmitting plate 1, on whicharranged is a diffraction grating area in a checkerboard pattern. Asshown in FIG. 3A, the diffraction grating area of the light-transmittingplate 1 includes a plurality of basic arrangements 1AB which arearranged two-dimensionally. Each of those basic arrangements 1ABconsists of four regions which are arranged in two columns and two rows.In each basic arrangement 1AB, a diffraction region 1D1 which forms partof the diffraction grating is arranged at the row 1, column 1 position,transparent regions 1CLR which form parts of a transparent member arearranged at the row 1, column 2 position and row 2, column 1 position,and another diffraction region 1D2 which forms part of the diffractiongrating is arranged at the row 2, column 2 position.

This light-transmitting plate 1 may have a size of 10 mm to 30 mm indiameter, and each basic arrangement 1AB may have a size of 100 μmsquare to 1000 μm square, for example. However, these sizes are just anexample, and do not have to be adopted as long as the functions to bedescribed later are realized.

The diffraction region 1D1 is a blazed grating which is designed so asto tilt the incoming light γ degrees with respect to the horizontaldirection (x direction). The diffraction region 1D2 is also a blazedgrating which is designed so as to tilt the incoming light −γ degreeswith respect to the horizontal direction. In addition, polarizationfilters (which form parts of a polarization area) are stacked on thesediffraction regions 1D1 and 1D2. The transparent regions 1CLR may bemade of a transparent member such as glass, plastic or cellophane.

FIG. 3B is a cross-sectional view as viewed on the plane A-A′ shown inFIG. 3A, and FIG. 3C is a cross-sectional view as viewed on the planeB-B′ shown in FIG. 3A. As shown in FIG. 3B, a polarization filter 1P1 isstacked on the back of the diffraction region 1D1 (i.e., to face theimage sensor 2). Likewise, as shown in FIG. 3C, a polarization filter1P2 is stacked on the back of the diffraction region 1D2. Thepolarization transmission axis directions (i.e., polarizationdirections) of these polarization filters 1P1 and 1P2 are different fromeach other by 90 degrees. In FIGS. 3B and 3C, the diffraction regions1D1 and 1D2 are illustrated as having a rectangular cross section forthe sake of simplicity. Actually, however, such a blazed grating has asaw-tooth cross section. By designing the shape appropriately,diffracted light can be output toward a particular direction.

Optionally, contrary to the configuration shown in FIGS. 3B and 3C, thediffraction regions 1D1 and 1D2 may be arranged on the back of thepolarization filters 1P1 and 1P2, respectively.

In FIGS. 3B and 3C, illustrated is an exemplary light ray which isincident perpendicularly onto, and gets diffracted by, the diffractionregion 1D1, 1D2. In this embodiment, the diffraction regions 1D1 and 1D2are designed so as to produce mostly a first-order diffracted light raywith almost no diffracted light rays of other orders. That is why thefirst-order diffracted light ray goes out in the direction indicated bythe arrow in FIGS. 3B and 3C. It should be noted that the diffractionregions 1D1 and 1D2 do not have to have such a property but may producediffracted light rays of other orders as well. Alternatively,second-order diffracted light rays or light rays of any other order maygo out mostly from the diffraction regions 1D1 and 1D2, instead of thefirst-order diffracted light rays.

FIG. 4 is a plan view illustrating a basic pixel arrangement for theimage sensor 2. As shown in FIG. 4, the image sensor 2 of thisembodiment has a basic pixel arrangement including four photosensitivecells 2 a, 2 b, 2 c and 2 d that are arranged in two columns and tworows. A polarization filter 2P1 is arranged to face the photosensitivecell 2 a at the row 1, column 1 position, and a polarization filter 2P2is arranged to face the photosensitive cell 2 d at the row 2, column 2position. But no polarization filters are arranged over thephotosensitive cell 2 b at the row 1, column 2 position and over thephotosensitive cell 2 c at the row 2, column 1 position. Thepolarization filters 2P1 and 2P2 have polarization directions that aredifferent from each other by degrees. Also, the polarization directionsof these polarization filters 2P1 and 2P2 are different from those ofthe polarization filters 1P1 and 1P2 by the angle θ.

FIG. 5 is a plan view schematically illustrating a part of aphotosensitive cell array 20 comprised of multiple photosensitive cellsthat are arranged in columns and rows on the imaging area of the imagesensor 2 and an array of polarization filters which are arranged overthe photosensitive cell array 20. Each of those photosensitive cellstypically includes a photodiode, which performs photoelectric conversionand outputs a photoelectrically converted signal representing thequantity of the light received. In this embodiment, the photosensitivecells 2 a through 2 d which are arranged in two columns and two rows andpolarization filters 2P1 and 2P2 which face them form a single unitelement, and a plurality of such unit elements are arrangedtwo-dimensionally on the imaging area.

By adopting such a configuration, the light that has entered this imagecapture device during exposure passes through the light-transmittingplate 1, the lens 3 and the infrared cut filter 4, gets imaged on theimaging area of the image sensor 2, and then is photoelectricallyconverted by the photosensitive cells 2 a through 2 d. And based on thephotoelectrically converted signals thus obtained, the image generatingsection 7 a of the image processing section 7 generates an image.

FIG. 6A schematically illustrates what images will be obtained bycapturing. Specifically, in the example illustrated in FIG. 6A, a lockhas been captured, the solid line indicates an image produced by thelight that has come directly from the subject, and the dotted lineindicates images produced by the diffracted light. Thus, the resultantimage can be said to be a synthetic image of the image produced by thedirect light and the two images produced by the diffracted light. Inthis case, the “light produced by the direct light” refers herein to animage to be produced by the light that has passed through thetransparent regions 1CLR of the light-transmitting plate 1. On the otherhand, the “images produced by the diffracted light” refer herein to theimages to be produced by the first-order diffracted light rays that havegone out through the diffraction regions 1D1 and 1D2 of thelight-transmitting plate 1. In the following description, the imageproduced by the direct light will be sometimes referred to herein as a“direct light image” and the two images produced by the diffracted lightrays will be sometimes referred to herein as “diffracted light images”.

In the schematic representation shown in FIG. 6A, the diffracted lightimages have shifted horizontally (laterally on the paper) with respectto the direct light image. This shift has been caused because thediffracted light ray going out through the diffraction region 1D1 istilted in the +x direction and the diffracted light ray going outthrough the diffraction region 1D2 is tilted in the −x direction. Also,since the two diffraction regions 1D1 and 1D2 are configured so thatthose diffracted light rays will be incident on the imaging area atrespective points that are symmetric to the points of incidence of thelight rays that have been transmitted through the transparent regions1CLR, those two diffracted light images will appear symmetrically withrespect to the direct light image. The shift of the diffracted lightimages with respect to the direct light image depends on the distancefrom the light-transmitting plate 1 to the subject. That is to say, thelonger the distance to the subject, the greater the magnitude of shift.That is why by detecting the magnitude of horizontal shift of thediffracted light images with respect to the direct light image, thesubject distance can be obtained.

In this embodiment, information indicating the correlation between themagnitudes of the horizontal shift of the diffracted light images andthe subject distance is stored in advance as a table or a function on astorage medium such as the memory 30. By reference to that correlation,the depth information generating section 7 b of the image processingsection 7 can calculate the subject distance based on the magnitude ofshift. Optionally, the correlation may be obtained by performing testshooting sessions a number of times with the subject distance changed.

Next, signal processing according to this embodiment will be described.In the following description, the output signals obtained as a result ofthe photoelectric conversion by the photosensitive cells 2 a, 2 b, 2 cand 2 d will be identified by S2a, S2b, S2c and S2d, respectively. Inthis example, the incoming light is supposed to be non-polarized lightand the diffraction regions 1D1 and 1D2 are supposed to diffract theincoming light 100% for the sake of simplicity. Also, the quantities (orintensities) of light rays that would be incident on a singlephotosensitive cell through the transparent regions 1CLR and thediffraction regions 1D1 and 1D2, were it not for the polarizationfilters 1P1, 1P2, 2P1 and 2P2, will be identified herein by D0, D1 andD2, respectively, which are supposed to be quantities per unit area ofthe light-transmitting plate 1. Furthermore, the transmittance in asituation where non-polarized light enters the polarization filters 1P1,1P2, 2P1 and 2P2 will be identified herein by T1 and the transmittancein a situation where light that is polarized in the same direction asthe polarization direction of a polarization filters enters thatpolarization filter will be identified herein by T2. That is to say, thetransmittance in a situation where non-polarized light enters twopolarization filters with the same polarization direction is representedby T1×T2. T1 and T2 are real numbers that satisfy 0<T1<1 and 0<T2<1.

During the exposure, the light that has come from the subject istransmitted through the light-transmitting plate 1 first. Thetransparent regions 1CLR account for a half of the overall area of thelight-transmitting plate 1 and the diffraction regions 1D1 and 1D2account for a quarter of the overall area of the light-transmittingplate 1. Since the polarization filters 1P1 and 1P2 are stacked on thediffraction regions 1D1 and 1D2, respectively, the transmittance of thediffraction regions 1D1 and 1D2 is determined by that of thepolarization filters 1P1 and 1P2 and represented by T1. That is whylight, of which the quantity is proportional to D0/2+T1 (D1+D2)/4, isincident on the photosensitive cells 2 b and 2 c, over which nopolarization filters are arranged, and the photosensitive cells 2 b and2 c output photoelectrically converted signals, of which the level isproportional to this quantity.

As for the photosensitive cell 2 a on which the polarization filter 2P1is stacked, on the other hand, if non-polarized light has entered, thequantity of the light transmitted is limited to T1 times as large asthat of the incoming light due to the influence of the polarizationfilter 2P1. Actually, however, the light that enters the polarizationfilter 2P1 includes a light ray which has come from the subject and beentransmitted through the transparent regions 1CLR and of which thequantity is proportional to D0/2, a light ray which has been transmittedthrough the diffraction region 1D1 and the polarization filter 1P1 andof which the quantity is proportional to (D1×T1×cos²θ)/4, and a lightray which has been transmitted through the diffraction region 1D2 andthe polarization filter 1P2 and of which the quantity is proportional to(D2×T1×sin²θ)/4. Consequently, the photosensitive cell 2 a generates asignal which is proportional to(D0×T1)/2+(D1×T1×cos²θ×T1)/4+(D2×T1×sin²θ×T2)/4.

Likewise, as for the photosensitive cell 2 d on which the polarizationfilter 2P2 is stacked, if non-polarized light has entered, the quantityof the light transmitted is limited to T1 times as large as that of theincoming light due to the influence of the polarization filter 2P2.Actually, however, the light that enters the polarization filter 2P2includes a light ray which has come from the subject and beentransmitted through the transparent regions 1CLR and of which thequantity is proportional to D0/2, a light ray which has been transmittedthrough the diffraction region 1D1 and the polarization filter 1P1 andof which the quantity is proportional to (D1×T1×sin²θ)/4, and a lightray which has been transmitted through the diffraction region 1D2 andthe polarization filter 1P2 and of which the quantity is proportional to(D2×T1×cos²θ)/4. Consequently, the photosensitive cell 2 d generates asignal which is proportional to(D0×T1)/2+(D1×T1×sin²θ×T1)/4+(D2×T1×cos²θ×T2)/4.

If the constant of proportionality between the quantity of lightincident on each pixel of the image sensor 2 and the signal generated issupposed to be one, S2a, S2b (=S2c), and S2d can be represented by thefollowing Equations (1) to (3), respectively. Furthermore, theseEquations (1) to (3) can also be represented by Equation (4) using amatrix. In Equation (4), only the right side of Equations (1) to (3) ismultiplied by four.

$\begin{matrix}{{S\; 2\; a} = {T\; 1( {{\frac{1}{2}D\; 0} + {\frac{1}{4}T\; 2\; \cos^{2}\theta \times D\; 1} + {\frac{1}{4}T\; 2\sin^{2}\theta \times D\; 2}} )}} & (1) \\{{S\; 2\; b} = {{S\; 2c} = {{\frac{1}{2}D\; 0} + {\frac{1}{4}T\; 1 \times D\; 1} + {\frac{1}{4}T\; 1 \times D\; 2}}}} & (2) \\{{S\; 2\; d} = {T\; 1( {{\frac{1}{2}D\; 0} + {\frac{1}{4}T\; 2\sin^{2}\theta \times D\; 1} + {\frac{1}{4}T\; 2\cos^{2}\theta \times D\; 2}} )}} & (3) \\{\begin{pmatrix}{S\; 2a} \\{S\; 2b} \\{S\; 2d}\end{pmatrix} = {\begin{pmatrix}{2T\; 1} & {T\; 1T\; 2\cos^{2}\theta} & {T\; 1T\; 2\; \sin^{2}\theta} \\2 & {T\; 1} & {T\; 1} \\{2\; T\; 1} & {T\; 1T\; 2\sin^{2}\theta} & {T\; 1T\; 2\cos^{2}\theta}\end{pmatrix}\begin{pmatrix}{D\; 0} \\{D\; 1} \\{D\; 2}\end{pmatrix}}} & (4)\end{matrix}$

If both sides of Equation (4) are multiplied from the left by theinverse matrix of the 3×3 matrix of Equation (4), the following Equation(5) can be obtained:

$\begin{matrix}{\begin{pmatrix}{D\; 0} \\{D\; 1} \\{D\; 2}\end{pmatrix} = {\begin{pmatrix}{2T\; 1} & {T\; 1T\; 2\cos^{2}\theta} & {T\; 1T\; 2\; \sin^{2}\theta} \\2 & {T\; 1} & {T\; 1} \\{2\; T\; 1} & {T\; 1T\; 2\sin^{2}\theta} & {T\; 1T\; 2\cos^{2}\theta}\end{pmatrix}^{- 1}\begin{pmatrix}{S\; 2a} \\{S\; 2b} \\{S\; 2d}\end{pmatrix}}} & (5)\end{matrix}$

By using the 3×3 inverse matrix and pixel signals S2a to S2d in Equation(5), an image signal D0 representing the direct light that has come fromthe subject and image signals D1 and D2 representing the diffractedlight can be obtained. That is to say, by using this Equation (5), theimage signals S2a to S2d that have been obtained by capturing can bedivided into an image represented by the direct light signal D0 and twoimages represented by the diffracted light signals D1 and D2. In thefollowing description, images based on these signals D0, D1 and D2 willbe sometimes referred to herein as a “D0 image”, a “D1 image” and a “D2image”, respectively, for the sake of simplicity.

These processing steps are carried out by the image generating section 7a of the image processing section 7. The photoelectrically convertedsignal that has been output from each photosensitive cell is sent to thesignal processing section 200 via the signal generating and receivingsection 5 and passed to the image generating section 7 a. In response,the image generating section 7 a performs the arithmetic processingdescribed above, thereby calculating the direct light signal D0 anddiffracted light signals D1, D2 on a unit element basis and generating adirect light image and two diffracted light images.

As can be seen, according to this embodiment, the direct light image andthe two diffracted light images can be separated from each other. Thus,the subject's depth information can be obtained by using these images.In addition, the D0 image generated based on the direct light may beprocessed as an ordinary image with a minimized decrease in resolution.

The depth information is calculated by the depth information generatingsection 7 b of the image processing section 7. In this embodiment, theimage to be formed by the direct light signal D0 and the two images tobe formed by the diffracted light signals D1 and D2 are treated aspseudo-color images. More specifically, the D0, D1 and D2 images aretreated as green (G), red (R) and blue (B) images, respectively, andwhite balance and colorization processing is carried out on these colorimages. And the D1 and D2 images are shifted toward the D0 image (i.e.,horizontally) so that the result of the colorization becomes closer tothe color white, and the magnitude of shift between the respectiveimages is detected. This is based on the principle that if the imagesbased on D0, D1 and D2 were the same image with no shift at all, nocoloring should be produced even if the white balance was struck and ifthe colorization processing was carried out.

Hereinafter, a specific example of the processing of obtaining depthinformation by calculating the magnitudes of shift of the D1 and D2images with respect to the D0 image will be described. In the example tobe described below, the image shown in FIG. 6A is supposed to beobtained as a result of the image capturing session.

First of all, the depth information generating section 7 b generatescolor images using the D0, D1 and D2 images as green (G), red (R) andblue (B) images, respectively, and strikes a white balance. During thisprocessing, red-based and blue-based white balance coefficients α0 andβ0 are determined. This processing can be represented by the followingnumerical expressions (that are Equations (6) and (7)), in which D0(x,y), D1 (x,y), and D2 (x,y) represent signal values at the pixellocation (x,y) on the D0, D1 and D2 images, respectively, and Σrepresents the sum calculated for every pixel:

α0=ΣD1(x,y)/ΣD0(x,y)  (6)

β0=ΣD2(x,y)/ΣD0(x,y)  (7)

Next, using these white balance coefficients α0 and β0, Cr (x,y) and Cb(x,y) represented by the following Equations (8) and (9) are generatedas colors red and blue signals, respectively:

Cr(x,y)=D1(x,y)−α0×D0(x,y)  (8)

Cb(x,y)=D2(x,y)−β0×D0(x,y)  (9)

And the sum Cs of the absolute values of the signals Cr and Cb iscalculated by making the arithmetic operation given by the followingEquation (10):

Cs=Σ|Cr(x,y)|+Σ|Cb(x,y)|  (10)

The signal Cs is used as an index indicating the degrees of color shiftof the D1 and D2 images with respect to the D0 image. The depthinformation generating section 7 b shifts these two diffracted lightimages in the direction of the direct light image on a pixel by pixelbasis, and repeatedly performs the arithmetic operations represented byEquations (8) to (10) every time the images are shifted. This processingis carried out until Cs becomes minimum.

If Cs becomes minimum, it means that the shift between the three imagesis minimum. The depth information generating section 7 b defines thetotal number of pixels that have been shifted from the initial statethrough the state in which the color shift has become minimum to be themagnitude of shift of the D1 and D2 images. And based on that magnitudeof shift and by reference to the correlation information that has beenstored in advance as a table or a function in the memory 30, the depthinformation generating section 7 b determines the distance from theimage capture device to the subject (i.e., depth information).

FIG. 6B illustrates how this processing is still in progress. As can beseen from FIG. 6B, the diffracted light images have become closer to thedirect light image compared to the initial state. And FIG. 6Cillustrates how the color shift has become minimum, i.e., how the threeimages have been matched to each other almost completely. If theprocessing described above is carried out with the images such as theones shown in FIGS. 6A through 6C presented on the display device, theuser can see visually the degree of matching between these images. As aresult, the matching processing can be carried out more easily thanpreviously.

FIG. 7A is a flowchart showing the procedure of the depth informationcalculation processing described above. First of all, in Step S701, thedepth information generating section 7 b retrieves the D0, D1 and D2images that have been generated by the image generating section 7 a fromthe memory 30 and calculates the signal Cs represented by Equation (10).In this processing step, instead of processing the D0, D1 and D2 imagesas they are, the processing may be carried out only on their croppedimage area including the subject that is the object of depth estimation.In that case, the number of processing steps to perform can be reduced,too. Next, in Step S702, with the D1 and D2 images shifted toward the D0image by one pixel, the signal Cs is calculated again. Subsequently, inStep S703, the decision is made whether or not the Cs value hasdecreased from the previous one. If the answer is YES, the processingstep S702 is performed all over again and the same calculation is madeonce again with the D1 and D2 images inched toward the D0 image by onemore pixel. And this series of processing steps is carried out over andover again until it turns out in Step S703 that the Cs value has startedto increase (i.e., until the decision is made that the Cs value hasreached its minimum value). If the Cs value has started to increase, theprocess advances to Step S704, in which the depth information generatingsection 7 b generates subject's depth information based on the totalnumber of pixels shifted so far and by reference to the correlationinformation. In this case, the “depth information” is a piece ofinformation indicating the distance from the image capture device to thesubject at the time of shooting and may be a numerical value or signrepresenting that distance.

FIG. 8 shows an exemplary relation between the number of pixels shifted(which will be referred to herein as “the pixel shift number”) and thesignal Cs (which will be referred to herein as “degree of R/Bcoloring”). In this example, the degree of R/B coloring becomes minimumwhen the number of pixels shifted is seven. Consequently, the shiftnumber of seven is defined to be the magnitude of shift.

By performing these processing steps, the depth information can beobtained. Even though the D1 and D2 images are supposed to be shifted byone pixel each time in the example described above, the number of pixelsshifted at a time does not have to be one but may also be two or more.Also, in the example described above, the depth information generatingsection 7 b automatically performs the processing of shifting the D1 andD2 images over and over again until Cs becomes minimum. However, thisprocessing may also be performed in accordance with a user'sinstruction. For example, the user may specify the number of pixelsshifted and the results may be presented on the display screen.Alternatively, the user may perform this operation of gradually shiftingthe D1 and D2 images toward the D0 image by him- or herself whilechecking out the monitor screen, and the results may be sequentiallypresented on the display screen. In that case, if the depth informationgenerating section 7 b is configured to indicate that the Cs value hasbecome minimum as soon as that happens and to present the magnitudes ofshift of the diffracted light images and the magnitude of depthcalculated on the screen, such an image capture device would come invery handy.

In the processing described above, the signal Cs in Equation (10)represents the magnitude of color shift of the overall image, and doesnot accurately represent the magnitude of shift of a portion of theimage including a particular subject. That is why the magnitude of shiftof a particular subject could not be determined accurately just byperforming the processing described above. For that reason, the depthinformation generating section 7 b of this embodiment may furtherperform the following additional processing.

FIG. 7B is a flowchart showing the procedure of this additionalprocessing. If the answer to the query of the processing step S703 shownin FIG. 7A is NO, the depth information generating section 7 b dividesin Step S801 each of the D0, D1 and D2 images by M horizontally and by Nvertically (where M and N are integers that are equal to or greater thantwo), thereby forming M×N partial images. Each image may be divided intoapproximately 100×100 partial images. In the following description, thepartial images of the D0, D1 and D2 images will be identified herein byzD0 (x,y), zD1 (x, y) and zD2 (x,y), respectively, where z indicates thenumber of the partial image and z=1 to M×N.

Next, in Step S802, using the white balance coefficients α0 and β0 thathave already been calculated, the depth information generating section 7b chooses a combination of partial images with the smallest degree ofcoloring while changing the combinations of the D0, D1 and D2 images oneafter another. Specifically, first of all, the depth informationgenerating section 7 b performs the same arithmetic operations as theones represented by Equations (8) and (9) on the chosen combination ofthe partial images. That is to say, ijCr (x,y) and ikCb (x,y) given bythe following Equations (11) and (12) are generated as color signals. InEquations (11) and (12), i, j and k are positive integers, the number ofthe partial image z=i, j, k, and the partial images of the D0, D1 and D2images are identified by iD0 (x, y), jD1 (x,y) and kD2 (x,y),respectively.

ijCr(x,y)=jD1(x,y)−α0×iD0(x,y)  (11)

ikCb(x,y)=kD2(x,y)−β0×iD0(x,y)  (12)

And the sum ijkCs of the absolute values of the signals ijCr and ikCb isobtained by making the arithmetic operation represented by the followingEquation (13). It should be noted that the summation Σ in Equation (13)is carried out on every pixel included in the partial images:

ijkCs=Σ|ijCr(x,y)|+Σ|ikCb(x,y)|  (13)

The depth information generating section 7 b calculates ijkCs with thecombinations (i,j,k) changed and chooses a combination of the partialimages that minimizes its value. As to the combination of partial imagesthat has been chosen in this manner, the magnitude of shift in the xdirection of the partial image of the D1 or D2 image with respect tothat of the D0 image is added to the magnitude of shift that has beenobtained in advance, and the sum is regarded as the final magnitude ofshift.

Next, in Step S803, the depth information generating section 7 bperforms the same arithmetic operations as what is represented byEquations (6) and (7) all over again on the chosen combination ofpartial images, thereby determining white balance coefficients α0 andβ0. The coefficients α0 and β0 that have been obtained previously havebeen determined with respect to the entire image, and are not idealcoefficients. That is to say, the D1 or D2 image has shifted either tothe right or to the left with respect to the D0 image, and either theleft- or right-end portion of the D1 or D2 image does not exist in theD0 image. That is why α0 and β0 that have been obtained previously doinvolve errors. Even if a synthetic image is generated using non-idealwhite balance coefficients α0 and β0, such a synthetic image does notexactly have the color white. For that reason, the depth informationgenerating section 7 b obtains more accurate white balance coefficientsα0 and β0 based on the partial images chosen. Since the partial imagesthat have been cropped out of D0, D1 and D2 are all related to the samesubject portion, α0 and β0 obtained from these partial images are moreideal coefficients than the one that have been obtained previously.

Subsequently, in the next processing step S804, the depth informationgenerating section 7 b calculates Cs all over again with respect to theentire image by Equation (10) using the white balance coefficients α0and β0 that have been determined once again. If it turns out, in StepS805, that the result of the calculation is smaller than a predeterminedthreshold value, the decision is made that it is “OK”. On the otherhand, if the result of the calculation turned out to be equal to orgreater than the threshold value, then the decision is made that it is“NG”. If the decision is “OK”, the depth information generating section7 b adds the magnitude of shift in the x direction of the partial imageof the D1 or D2 image with respect to that of the D0 image in Step S806to the magnitude of shift that has been obtained in advance, and regardsthe sum as the final magnitude of shift. Then, in Step S807, the depthinformation generating section 7 b calculates the depth information ofthe subject of interest based on that magnitude of shift.

If the decision made in Step S805 is “NG”, then the depth informationgenerating section 7 b displays in Step S808 a message saying that thedepth information cannot be obtained on the screen. Optionally, insteadof displaying such a message on the screen, distance information basedon the total number of pixels that has been obtained in Step S704 shownin FIG. 7A may be output as well.

In this manner, the depth information generating section 7 b performscolor image processing and determines depth information using threeimages. By obtaining the magnitude of shift of the diffracted lightimage with respect to the direct light image, the subject's relativedepth information can be calculated.

As can be seen, according to this embodiment, by providing an opticalregions (i.e., the light-transmitting plate 1) in which diffractiongratings 1D1 and 1D2 and polarization filters 1P1 and 1P2 are stackedone upon the other for the image capturing optical system and byarranging pixels with polarization filters 2P1 and 2P2 on the imagesensor 2, too, an image produced by the light that has come directlyfrom the subject and an image produced by diffracted light can becaptured so as to superpose one upon the other. And by performingarithmetic processing between pixels, those images can be separated fromeach other. Furthermore, by performing white balance and colorizationprocessing on an appropriate combination of partial images that havebeen cropped out of those separated images, the magnitude of shiftbetween the image produced by the direct light and the image produced bythe diffracted light can be calculated. As a result, the subject's depthinformation can be obtained.

In the embodiment described above, the light-transmitting plate 1 issupposed to be configured so that the diffraction regions 1D1 and 1D2are arranged all over the light-transmitting plate 1 as shown in FIG.3A. However, this is just an example. Alternatively, the diffractionregions may be divided into two connected areas 1 a as shown in FIG. 9A.Still alternatively, the diffraction regions may also be divided intothree or more separate areas 1 a as shown in FIG. 9B. In those cases,each of those areas 1 a is comprised of multiple basic arrangements 1ABas in the light-transmitting plate 1 shown in FIG. 3A and the rest ofthe light-transmitting plate 1 a other than the areas 1 a is transparentregions. If such a configuration is adopted, the quantity of thediffracted light decreases but the quantity of the direct lightincreases compared to the configuration shown in FIG. 3A. That is whysuch a configuration is suitably used when an ordinary image needs to beobtained with high sensitivity. It should be noted that even in such anexample, the basic arrangement 1AB of the light-transmitting plate 1does not have to be a matrix of two rows by two columns, either.

Furthermore, in the embodiment described above, two polarizationfilters, of which the polarization directions intersect with each otherat right angles, are supposed to be used as the polarization filters 1D1and 1D2. However, the polarization directions of the polarizationfilters do not have to intersect with each other at right angles as longas the relations similar to the ones represented by Equations (4) and(5) are satisfied. The same can be said about the polarization filters2D1 and 2D2 of the image sensor 2. That is to say, their polarizationdirections do not have to define right angles, either. If the angledefined by the polarization transmission axes of the polarizationfilters 1D1 and 1D2 of the light-transmitting plate 1 is represented byφ1 and if the angle defined by the polarization transmission axes of thepolarization filters 2D1 and 2D2 of the image sensor 2 is represented byφ2 by generalizing the configuration described above, Equations (4) and(5) may be modified into the following Equations (14) and (15),respectively:

$\begin{matrix}{\begin{pmatrix}{S\; 2a} \\{S\; 2b} \\{S\; 2d}\end{pmatrix} = {\begin{pmatrix}{2T\; 1} & {T\; 1T\; 2\cos^{2}\theta} & {T\; 1T\; 2\; {\cos^{2}( {\theta - {\varphi \; 1}} )}} \\2 & {T\; 1} & {T\; 1} \\{2\; T\; 1} & {T\; 1T\; 2{\cos^{2}( {\theta + {\varphi \; 2}} )}} & {T\; 1T\; 2{\cos^{2}( {\theta + {\varphi 2} - {\varphi 1}} )}}\end{pmatrix}\begin{pmatrix}{D\; 0} \\{D\; 1} \\{D\; 2}\end{pmatrix}}} & (14) \\{\begin{pmatrix}{D\; 0} \\{D\; 1} \\{D\; 2}\end{pmatrix} = {\begin{pmatrix}{2T\; 1} & {T\; 1T\; 2\cos^{2}\theta} & {T\; 1T\; 2\; {\cos^{2}( {\theta - {\varphi \; 1}} )}} \\2 & {T\; 1} & {T\; 1} \\{2\; T\; 1} & {T\; 1T\; 2{\cos^{2}( {\theta + {\varphi \; 2}} )}} & {T\; 1T\; 2{\cos^{2}( {\theta + {\varphi 2} - {\varphi 1}} )}}\end{pmatrix}^{- 1}\begin{pmatrix}{S\; 2a} \\{S\; 2b} \\{S\; 2d}\end{pmatrix}}} & (15)\end{matrix}$

Since the angles θ, φ1 and φ2 are already known, the image generatingsection 7 a can generate the D0, D1 and D2 images by performing thearithmetic operation based on Equation (15) on each and every unitelement of the image sensor 1.

In addition, each of the light-transmitting plate 1 and the image sensor2 may have only one kind of polarization filters, instead of the twokinds of filters. In that case, the light-transmitting plate 1 may haveonly a single kind of diffraction regions. Nevertheless, in that case,the 3×3 matrix of Equation (4) needs to be replaced with a 2×2 matrix.Suppose, as a simple example, a situation where the polarization filters1P2 and 1P1 have the same property in FIG. 3A and the polarizationfilters 2P2 and 2P1 have the same property in FIG. 4A. In that case, inthe image sensor 2, the processing can be carried out on the basis ofpixels in one row by two columns or in two rows by one column, andEquations (4) and (5) can be rewritten into the following Equations (16)and (17), respectively:

$\begin{matrix}{\begin{pmatrix}{S\; 2\; a} \\{S\; 2b}\end{pmatrix} = {\begin{pmatrix}{T\; 1} & {T\; 1T\; 2\cos^{2}\theta} \\1 & {T\; 1}\end{pmatrix}\begin{pmatrix}{D\; 0} \\{D\; 1}\end{pmatrix}}} & (16) \\{\begin{pmatrix}{D\; 0} \\{D\; 1}\end{pmatrix} = {\begin{pmatrix}{T\; 1} & {T\; 1T\; 2\cos^{2}\theta} \\1 & {T\; 1}\end{pmatrix}^{- 1}\begin{pmatrix}{S\; 2\; a} \\{S\; 2b}\end{pmatrix}}} & (17)\end{matrix}$

In this case, the image generating section 7 a can obtain an imageproduced by the direct light and an image produced by the diffractedlight by calculating the signals D0 and D1 by Equation (17). When thecolorization processing is carried out after that, there is no problemif the diffracted light image is used as a magenta (that is the mixtureof the colors red and blue) image. In that case, the depth informationgenerating section 7 b may calculate the signal Cs represented byEquation (10) while shifting the D1 image that is produced by thediffracted light horizontally toward the D0 image with D1 (x,y)=D2(x,y), α0=β0 and Cr (x,y)=Cb (x,y) satisfied in Equations (6) to (9).The same can be said about the processing shown in FIG. 7B, too.

Also, as for the polarization filters of the image sensor 2, their basicarrangement does not have to have the checkerboard pattern but may alsobe a vertical striped arrangement, a horizontal striped arrangement, orany other suitable arrangement. For example, the polarization filters2P1 and 2P2 shown in FIG. 4 may be arranged either on the same row or onthe same column.

In the embodiment described above, the diffraction regions 1D1 and 1D2of the light-transmitting plate 1 are supposed to generate mainlyfirst-order diffracted light rays. However, the diffraction regions 1D1and 1D2 may also be configured to generate mainly diffracted light raysof any other order, too. In this description, if a diffraction region“mainly generates n^(th)-order diffracted light rays (where n is aninteger that is equal to or greater than one)”, then it means thatn^(th)-order diffracted light rays account for 80% or more of theoverall diffracted light going out of that diffraction region. Eachdiffraction region is designed so that the n^(th)-order diffracted lightrays suitably account for 90% or more, more suitably 95% or more, of theoverall diffracted light.

Furthermore, in the embodiment described above, the diffraction regions1D1 and 1D2 are configured so that the point of incidence on the imagingarea of the n^(th)-order diffracted light ray generated by each of thoseregions shifts in the x direction with respect to the point of incidenceof the light ray that has been transmitted through the transparentregion 1CLR. However, this is only an example of the present invention.Alternatively, the diffraction regions 1D1 and 1D2 may also beconfigured so that the point of incidence of the n^(th)-order diffractedlight ray on the imaging area shifts in the y direction or obliquelywith respect to that of the directly incident light.

Furthermore, even though the D0, D1 and D2 images are supposed to begreen, red and blue images, respectively, in the embodiment describedabove, the colors have been assigned in this manner just for conveniencesake, and a combination of any other colors may also be used. As long asthose images are treated as images in different colors, any othercombination of colors may be adopted as well. For example, processingmay also be carried out with the D0 image supposed to be not a greenimage but a red image and with the D1 and D2 images supposed to be blueand green images, respectively.

Embodiment 2

Hereinafter, a depth estimating image capture device as a secondembodiment of the present invention will be described. This embodimenthas the same configuration as the first embodiment except theconfiguration of the light-transmitting plate 1. Thus, the followingdescription of this second embodiment will be focused on differencesfrom the first embodiment, and their common features will not bedescribed all over again.

FIG. 10A is a plan view illustrating a light-transmitting plate 1according to this embodiment. FIG. 10B is a cross-sectional view thereofas viewed along the plane C-C′ shown in FIG. 10A. Even though the basicarrangement 1CC of the diffraction grating regions is also supposed tohave a checkerboard pattern according to this embodiment, this is not anessential requirement. In the basic arrangement 1CC of thelight-transmitting plate 1, diffraction regions 1D4 are arranged at therow 1, column 1 and row 2, column 2 positions and transparent regions1CLR are arranged at the row 1, column 2 and row 2, column 1 positions.The diffraction regions 1D4 are obtained by cutting linear grooves on atransparent member, and will be referred to herein as “lineardiffraction gratings”. These diffraction gratings are designed so as totilt the incoming light ±γ degrees with respect to the horizontaldirection. In FIG. 10B, the light that has come directly from thesubject is illustrated as “zero-order light rays”, light rays whichdefine a tilt angle of γ degrees with respect to the direction of thedirect light are illustrated as “+first-order light rays”, and lightrays which define a tilt angle of −γ degrees with respect to thedirection of the direct light are illustrated as “−first-order lightrays”. Optionally, the diffraction regions 1D4 may also be configured togenerate mainly diffracted light rays of any other order instead of the±first-order light rays.

In this linear diffraction region 1D4, polarization filters 1P1 and 1P2are stacked one upon the other so as to cover the diffraction region 1D4only partially. The polarization filters 1P1 and 1P2 are arranged wherethe first-order light rays transmitted through the linear diffractionregions 1D4 pass. The respective polarization directions of thepolarization filters 1P1 and 1P2 are supposed in this embodiment to, butdo not have to, intersect with each other at right angles. As can beseen, according to this embodiment, two polarization filters 1P1 and 1P2are arranged for a single groove on the diffraction region 1D4. As aresult, the ±first-order light rays get polarized but the zero-orderlight ray is not polarized but transmitted as it is.

The linear diffraction region 1D4 for use in this embodiment alsotransmits the zero-order light ray that is the light coming directlyfrom the subject. That is why compared to the configuration of the firstembodiment, the levels of the signals of the photosensitive cells 2 aand 2 d decrease but the levels of the signals of the photosensitivecells 2 b and 2 c increase. This is equivalent to changing the arearatio of the transparent regions 1CLR and diffraction regions 1D1 and1D2 in the light-transmitting plate 1 of the first embodiment, and isjust a matter of design. That is why if those design values areintroduced into Equation (4), the subject's depth information can beobtained by performing quite the same processing as in the firstembodiment.

As can be seen, according to this embodiment, by adopting an imagecapturing optical system of the same configuration and performing thesame signal processing as in the first embodiment and by using thelight-transmitting plate 1 of which the diffraction grating is a linearone and where polarization filters are arranged on diffracted lighttransmitting regions, the subject's depth information can be calculatedas in the first embodiment described above.

In the embodiment described above, the light-transmitting plate 1 issupposed to be configured so that the diffraction regions 1D4 arearranged as shown in FIG. 10A. However, this is just an example.Alternatively, the diffraction regions may be divided into two connectedareas 1 a as shown in FIG. 9A. Still alternatively, the diffractionregions may also be divided into three or more separate areas 1 a asshown in FIG. 9B. The basic arrangement of the light-transmitting plate1 and the image sensor 2 does not have to be a matrix of two rows by twocolumns. Furthermore, the polarization directions of the two kinds ofpolarization filters to use do not have to intersect with each other atright angles as long as a relation similar to the one represented byEquation (4) is satisfied. The same can be said about the polarizationfilters of the image sensor 2. That is to say, their basic arrangementdoes not have to have a checkerboard pattern but may also be a verticalstriped arrangement or horizontal striped arrangement as well. The samestatement as what has already been described for the first embodimentalso applies to other modified examples.

Embodiment 3

Next, a third embodiment will be described. This embodiment relates toan image processor with no image capturing system. In the firstembodiment described above, the image capture device is supposed toperform the image generation processing, matching the direct light imageand the diffracted light images to each other, and calculate themagnitude of shift of the diffracted light images with respect to thedirect light image by itself. However, such processing may be carriedout by another device, not the image capture device itself. In thatcase, the image capture device itself does not have to include the imageprocessing section 7 shown in FIG. 1.

The image processor of this embodiment has the same configuration as thesignal processing section 200 shown in FIG. 1. The image processor maybe built in the image capture device or may also be implemented as acomputer such as a personal computer or a mobile telecommunicationsdevice.

The image processor receives pixel signals that have been generated byan image capture device with the configuration shown in FIGS. 2 to 5 orthe configuration shown in FIGS. 10A and 10B, divides them into the D0,D1 and D2 images, and then performs the depth estimation processingshown in FIG. 7A, for example. Alternatively, the image processor mayalso perform the processing steps S701 through S703 shown in FIG. 7A andthe processing steps shown in FIG. 7B. In this manner, the subject'sdepth information can be generated.

It should be noted that the image processing of this embodiment isapplicable to not only images that have been obtained by an imagecapture device with the image capturing section 100 of the first orsecond embodiment but also any other arbitrary images that should bematched such as a plurality of images obtained by shooting the samesubject from multiple different viewpoints. Particularly when aplurality of images with mutually different luminance levels (orcontrasts) need to be matched, it is difficult to get that matching doneaccurately by a known method but even such matching can get done easilyand accurately according to this embodiment. Since the processing ofthis embodiment is carried out with mutually different colors assignedto multiple images, the user can visually sense the degrees of matchingbetween those images by performing such processing with those imagespresented on the display. As a result, the matching processing can getdone much more easily than previously. Optionally, such an imageprocessor may be configured to provide information indicating themagnitude of shift between multiple images or a matched synthetic imagewithout generating depth information.

Hereinafter, a more generalized one of the configuration of the imageprocessor according to this embodiment will be described.

FIG. 11 is a block diagram illustrating a general configuration for sucha generalized image processor 10. The image processor 10 includes aninput interface (IF) 11 which accepts first and second images, an imageprocessing section 13 which performs matching processing on those imagesaccepted, and an output interface (IF) 12 which outputs the result ofthe processing to an external device such as a display or a storagemedium. The image processing section 13 includes a memory 30 whichstores necessary information about various kinds of parameters orprograms involved with the image processing, a color image generatingsection 13 a which generates a pseudo-synthetic color image based on thefirst and second images that have been input, a decision section 13 bwhich calculates an index value indicating the degree of color shift ofthe synthetic color image to determine the degree of matching betweenthe first and second images, and an image moving section 13 c whichmoves the second image toward the first image based on the result of thedecision made by the decision section 13 b. The image processing section13 is suitably implemented as a combination of a hardware component suchas a DSP or a CPU and a software program. Alternatively, the imageprocessing section 13 may also be implemented as a dedicated integratedcircuit that can perform the functions of these sections.

The input interface 11 is an interface which gets first and secondimages representing the same subject. The input interface 11 is eitherhardwired or connected wirelessly to another device (such as an imagecapture device or a mobile telecommunications device) and can obtainfirst and second images that should be matched to each other from thatdevice.

As long as the first and second images are two images which representthe same subject but in one of which the position of the subject hasshifted in a particular direction with respect to the position in theother, the first and second images may have been obtained by any method.For instance, the direct light image and diffracted light imagesobtained by the image capture device of the first or second embodimentor the image capture device disclosed in Patent Document No. 9 areexamples of the first and second images. Alternatively, the first andsecond images may also be two images with parallax (i.e., stereoscopicimages) which have been obtained by shooting the same subject frommultiple different viewpoints. In any case, each of these pairs ofimages is characterized in that the same subject is represented atdifferent positions (i.e., at different sets of coordinates) and thatthose images have different overall luminance levels. It is not easy toget those two images accurately matched to each other by a conventionalmethod. However, the image processing section 13 of this embodiment canget the matching processing done visually by processing those images aspseudo-color images.

The color image generating section 13 a generates a color image whichuses the pixel values (luminance values) of the first and second imagesas first and second color values, respectively. For example, if therespective pixels of a color image are represented by the values of thethree primary colors of red (R), green (G) and blue (B), the first colormay be the color green and the second color may be the color magenta(i.e., mixture of the colors red and blue) which is the complementarycolor of the color green. If such a combination of colors is adopted,the color image to be obtained after matching will be presented in thecolor white, and therefore, will have an increased degree of visibility.It should be noted that the first and second colors may be any othercolors as long as they are two different colors. For example, one of thefirst and second colors may be one of the colors red, green and blue andthe other may be the complementary color of the former color. Or thefirst and second colors may be two colors chosen from the colors red,green and blue. In the latter case, in the synthesized color image, amatched portion will be presented not in the color white but in amixture of the two chosen colors.

The decision section 13 b calculates an index value indicating thedegree of color shift between the first and second colors of the colorimage that has been generated by the color image generating section 13 aand determines the degree of matching between the first and secondimages based on that index value. For example, this index valuecorresponds to Cs of Equation (10) that has already been described forthe first embodiment. As the color image is composed of two colors inthis embodiment, the first and second terms on the right side ofEquation (10) are processed as having the same value. The decisionsection 13 b of this embodiment adjusts the balance between the valuesof the first and second colors of the color image, and obtains the indexvalue by performing an arithmetic operation including calculating thedifference between the first and second color values. Optionally, aconfiguration in which the first and second color values are directlycompared to each other without adjusting the balance between the firstand second color values may be adopted.

If the decision is made by the decision section 13 b that the first andsecond images do not match each other, the image moving section 13 cperforms the processing of replacing the pixel value of each pixel ofthe second image with the pixel value of an adjacent pixel. That is tosay, the image moving section 13 c slides the second image so that theposition of the subject on the second image becomes closer to that ofthe same subject on the first image. In this case, the sliding directionjust needs to agree with the direction of shift between the first andsecond images, and may be not only the horizontal direction but also anyother arbitrary direction as well. The image moving section 13 c doesnot have to move the entire second image pixel by pixel, but may alsomove it two or more pixels at a time as well.

FIG. 12 is a schematic representation illustrating conceptually how toget the processing of this embodiment done. In the example illustratedin FIG. 12, the subject represented in the first image is located atapproximately the center of the image, but the subject represented inthe second image is located on the right-hand side above the center. Acolor image, in which the respective pixel values of the first andsecond images are used as first and second color values, is as shown atthe middle portion of FIG. 12. By getting the processing done by thedecision section 13 b and the image moving section 13 c repeatedly onthis state as an initial state, the image shown at the bottom portion ofFIG. 12 can be obtained.

FIG. 13 is a flowchart showing the procedure of the processing to getdone by the image processor 10. First of all, in Step S101, the inputinterface 11 gets the first and second images. Next, in Step S102, thecolor image generating section 13 a generates a color image in which therespective pixel values of the first and second images are used as firstand second color component values, respectively. Subsequently, in StepS103, the decision section 13 b calculates an index value indicating thedegree of color shift between the first and second colors. Then, in StepS104, the decision section 13 b determines the degree of matchingbetween the first and second images based on that index value. In thiscase, if the decision has been made that the first and second images donot match each other, then the image moving section 13 c replaces thepixel value of each pixel of the second image with that of an adjacentpixel in Step S105. That is to say, the image moving section 13 c slidesthe second image toward the first image. After that, the process goesback to the processing step S102 and the same series of processing stepsare performed over and over again until the decision is made that thefirst and second images match each other.

In Step S104, the degree of matching can be determined in the same wayas in the first embodiment. That is to say, if the index value isgreater than the previous value, the decision section 13 b can make adecision that the index value is minimum and that the first and secondimages now match each other.

Optionally, after the decision has been made in Step S104 shown in FIG.13 that the two images match each other, the magnitude of slight shiftmay be obtained. Such processing corresponds to the processing of thefirst embodiment shown in FIG. 7B. In that case, as shown in FIG. 14,the image processing section 13 further includes an image dividingsection 13 d which divides each of those two images into a plurality ofpartial images. After the processing step S104 has been performed, theimage dividing section 13 d divides each of the first and second imagesinto a plurality of partial images. While changing the combinations ofpartial images which have been selected one from the first image and theother from the second image, the decision section 13 b calculates anindex value for a region associated with the combination of the partialimages, and chooses a combination of partial images with the highestdegree of matching. The image moving section 13 c performs theprocessing of making the second image even closer to the first imagebased on a difference in coordinate between the partial images in thecombination that has been chosen by the decision section 13 b. Byperforming these processing steps, the degree of matching can bedetermined even more accurately.

After having performed this matching processing, the image processingsection 13 may output either information indicating the magnitude ofshift between the two images or the synthetic color image subjected tothe matching processing to an external storage medium or display sectionvia the output interface section 12. Optionally, the image processingsection 13 may be configured to present sequentially, on the display,color images that have been generated one after another during thematching processing. If the image processing section 13 has such aconfiguration, the user can see visually how the matching processing isgoing. Optionally, the image processing section 13 may also beconfigured to accept the user's instructions through the input interface11.

In the foregoing description, two images are supposed to be matched toeach other. However, the image processing section 13 may also beconfigured to match three images just like the image processing section7 of the first and second embodiments. In that case, a third imagerepresenting the same subject as the first and second images is furtherinput to the input interface. In this case, the position of the subjecton the third image and the position of the same subject on the secondimage are symmetric to the position of the subject on the first image.In such a situation, the color image generating section 13 a generates acolor image in which the respective pixel values of the first, secondand third images are used as first, second and third color values,respectively. And the decision section 13 b may be configured tocalculate an index value indicating the degree of color shift betweenthe first to third images and determine, based on the index value,whether or not the second and third images match the first image. If thedecision has been made that the second and third images do not match thefirst image, the image moving section 13 c performs the processing ofreplacing the pixel value of each pixel of the second and third imageswith that of a pixel which is adjacent in the direction toward the firstimage. By performing such processing, these three images can be matchedto each other.

As can be seen, the image processor of this embodiment can easily matcha plurality of images representing the same subject effectively. That iswhy this embodiment can be used not just to generate depth informationbut also in any application that ever needs matching multiple images toeach other.

INDUSTRIAL APPLICABILITY

A depth estimating image capture device according to an embodiment ofthe present invention can be used effectively in every type of camerasuch as a digital camera, a digital movie camera, a solid-state camerafor broadcasting, and an industrial solid-state surveillance camera.Also, an image processor according to an embodiment of the presentinvention can be used effectively to match not just the images capturedby the depth estimating image capture device described above but alsoimages captured by a stereoscopic image capture device or any other kindof image capture device as well.

REFERENCE SIGNS LIST

-   1 light-transmitting plate-   1AB, 1CC light-transmitting plate's basic arrangement-   1D1, 1D2, 1D4 diffraction region-   1P1, 1P2, 2P1, 2P2 polarization filter (polarization region)-   1CLR transparent region-   2 solid-state image sensor-   2 a, 2 b, 2 c, 2 d photosensitive cell-   3 optical lens-   4 infrared cut filter-   5 signal generating and receiving section-   6 sensor driving section-   7 image processing section-   7 a image generating section-   7 b depth information generating section-   8 interface section-   10 image processor-   11 input interface-   12 output interface-   13 image processing section-   13 a color image generating section-   13 b decision section-   13 c image moving section-   13 d image dividing section-   30 memory-   100 image capturing section-   200 signal processing section

1. An image processor that carries out matching on a plurality of imagesrepresenting the same subject, the processor comprising: an inputinterface that receives first and second images on one of which theposition of the subject has shifted in a particular direction from theposition on the other; a color image generating section that generates acolor image in which the pixel values of respective pixels of the firstand second images are used as the values of first and second colors,respectively; a decision section that calculates an index valueindicating the degree of color shift between the first and second colorsin the color image and that determines, based on the index value,whether or not the first and second images match each other; and animage moving section that performs, if the decision has been made thatthe first and second images do not match each other, the processing ofmaking the second image closer to the first image by replacing the pixelvalue of each pixel of the second image with the pixel value of a pixelthat is adjacent to the former pixel in the particular direction.
 2. Theimage processor of claim 1, wherein if the image moving section hasperformed the processing, the decision section determines again whetheror not the first and second images match each other, and wherein theimage moving section performs the processing over and over again untilthe decision is made by the decision section that the first and secondimages match each other.
 3. The image processor of claim 1, wherein thedecision section adjusts the balance between the respective values ofthe first and second colors in multiple pixels of the color image, andobtains the index value by performing an arithmetic operation includingcalculating the difference between the respective values of the firstand second colors that have been adjusted in each said pixel.
 4. Theimage processor of claim 3, wherein the index value is obtained byadding together either the absolute values, or the squares, of thedifferences between the respective values of the first and second colorsthat have been adjusted with respect to every pixel.
 5. The imageprocessor of claim 4, wherein if it has turned out, as a result of theprocessing by the image moving section, that the index value hasincreased from the previous one, the decision is made by the decisionsection that the first and second images match each other.
 6. The imageprocessor of claim 1, further comprising an image dividing section thatdivides, if the decision has been made that the first and second imagesmatch each other, each of the first and second images into a pluralityof partial images, wherein while changing combinations of the partialimages, one of which has been selected from the first image and theother of which has been selected from the second image, the decisionsection calculates the index value with respect to an area of the colorimage associated with the combination of the partial images, therebychoosing a combination of the partial images that have the highestdegree of matching, and wherein the image moving section makes thesecond image even closer to the first image based on a difference incoordinate between the partial images in the combination that has beenchosen by the decision section.
 7. The image processor of claim 1,wherein the first color is one of the colors red, green and blue and thesecond color is the complementary color of the first color.
 8. The imageprocessor of claim 1, wherein the input interface further obtains athird image representing the same subject as the first and secondimages, and if the respective positions of the subject on the second andthird images are symmetric to the position of the subject on the firstimage, the color image generating section generates the color image inwhich the respective pixel values of the first, second and third imagesare used as the values of the first, second and third colors,respectively, and the decision section calculates an index valueindicating the degree of color shift between the first, second and thirdcolors and determines, based on the index value, whether or not thesecond and third images match the first image, and if the decision hasbeen made that the second and third images do not match the first image,the image moving section performs the processing of making the secondand third images closer to the first image by replacing the pixel valueof each pixel of the second image with the pixel value of a pixel thatis adjacent to the former pixel in a first direction and by replacingthe pixel value of each pixel of the third image with the pixel value ofa pixel that is adjacent to the former pixel in a direction opposite tothe first direction.
 9. The image processor of claim 8, wherein thefirst, second and third colors are respectively one, another and theother of the colors red, green and blue.
 10. The image processor ofclaim 1, further comprising an output interface that outputs informationabout the magnitude of overall motion of the second image as a result ofthe processing by the image moving section.
 11. An image capture devicecomprising: the image processor of claim 1; and an image capturingsection that obtains the first and second images by capturing.
 12. Animage processing method for carrying out matching on a plurality ofimages representing the same subject, the method comprising the stepsof: receiving first and second images, on one of which the position ofthe subject has shifted in a particular direction from the position onthe other; generating a color image in which the pixel values ofrespective pixels of the first and second images are used as the valuesof first and second colors, respectively; calculating an index valueindicating the degree of color shift between the first and second colorsin the color image; determining, based on the index value, whether ornot the first and second images match each other; and if the decisionhas been made that the first and second images do not match each other,making the second image closer to the first image by replacing the pixelvalue of each pixel of the second image with the pixel value of a pixelthat is adjacent to the former pixel in the particular direction.
 13. Acomputer program, stored on a non-transitory computer readable storagemedium, for carrying out matching on a plurality of images representingthe same subject, the program being defined to make a computer performthe steps of: receiving first and second images on one of which theposition of the subject has shifted in a particular direction from theposition on the other; generating a color image in which the pixelvalues of respective pixels of the first and second images are used asthe values of first and second colors, respectively; calculating anindex value indicating the degree of color shift between the first andsecond colors in the color image; determining, based on the index value,whether or not the first and second images match each other; and if thedecision has been made that the first and second images do not matcheach other, making the second image closer to the first image byreplacing the pixel value of each pixel of the second image with thepixel value of a pixel that is adjacent to the former pixel in theparticular direction.